Using Vision to Guide an Hors d’Oeuvres Serving Robot

This paper describes the vision system used on the robot that took first place in both the technical and popular categories of the 1998 American Association of Artificial Intelligence Hors d’Oeuvres Anyone? competition. The robot used visual input to execute multiple tasks: 1) locate people to serve, 2) determine if someone was reaching for food from the robot’s tray, 3) determine if the person being served was a VIP or robot competition human participant, and 4) determine where its refill station was located. The robot successfully executed all of these tasks during the competition. All of these tasks were accomplished using a low-cost, low-quality color camera.